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Abstract 

Recent biological experimental findings have shown that synaptic plasticity de- 
pends on the relative timing of pre- and post-synaptic spikes and this is called 
spike-timing-dependent plasticity (STDP). Many authors have claimed that a 
precise balance between long-term potentiation (LTP) and long-term depression 
(LTD) of STDP is crucial in the storage of spatio-temporal patterns. Some au- 
thors have numerically investigated the impact of an imbalance between LTP and 
LTD on the network properties. However, the mathematical mechanism remains 
unknown. We analytically show that an associative memory network has the ro- 
bust retrieval properties of spatio-temporal patterns, and these properties make 
the network less vulnerable to any deviation from a precise balance between LTP 
and LTD when the network contains a finite number of neurons. 
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1 Introduction 



Recent biological experimental findings have indicated that synaptic plasticity 
depends on the relative timing of pre- and post-synaptic spikes. This relative 



sion (LTD) is induced (jBi and Poo 



1998; 



Markram et al 



or long- 


term depres- 




1997 




Zhang et al. 



1998). LTP occurs when presynaptic firing precedes postsynaptic firing by no 
more than about 20 ms. In contrast, LTD occurs when presynaptic firing fol- 
lows postsynaptic firing. A rapid transition between LTP and LTD takes place 



within a few milliseconds. A learning ru 



dependent plasticity f STDP) (ISong et a 
bian learning (TAH) ([Abbott and Song . 



e of this type is called spike-timing- 



1999: 



2000) or tempora 



Rubin et al- 



l y asym metric Heb- 



tional role of STDP has 



1996 



Kempter et al. 



1999: 



)een investigated 



Abbott and Song . 



jy many authors 



1999; 



2001) . The func- 



Gerstner et al 



Munro and Hernandez 



2000; 



Ra o and Sei nowski 



12000: ISong et all 1200(1 iRubin et all BOO it iLevv et al 



2001 



van Rossum et al 



2001 



Song and Abbott 



2001; 



Yoshioka, 



Karmarkar and Buonomand . 120 02; Ma tsumoto and Okada . 



2002; Fu et al. 



2002; 



2002). Some of these 



authors have claimed that the precise balance betwe en the LTP and LTD of STDP 



is crucial i n the storage of spatio-temporal patt terns (|Munro and Hernandez . 



Yoshioka 



2002; 



Matsumoto and Okada 



2000; 



2002). In our previous work, we analyti- 



cally showed that STDP has the sam e qualitative effect as the covariance rule when 



the balance is precisely maintained ((Matsumoto and Okada , 



20021) . In the brain, 



though, it is inconceivable that such a balance will be precisely maintained. Some 
authors numerically have investigated the impact of an imbalance between LTP 
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and LTD on ne twork properties (pong et al 



Yoshioka 



2000; 



Munro and Hernandez! . 



2000; 



2002). Munro and Hernandez showed that spatio-t emporal patterns 



cann ot be retrieved in a noisy environment when there is no LTD ((Munro and Hernandez , 



2000). However, the mathematical mechanism that accounts for this remains un- 
known. 

In this paper, we analytically investigate the retrieval properties of spatio- 
temporal patterns in an associative memory network where there is an imbal- 



ance between the LTP and LTD o 



neuro dynamics ( A mari and Maginu 



STD 



1988; 



by using a method of the statistical 



Okada . 



1996 



Matsumoto and Okada . 



2002). Using this theory, we discuss the macroscopic behavior of the network 
at a thermodynamic limit: the number of neurons is infinite. When the bal- 
ance is precisely maintained, sequential patterns can be stored in the network 



( Matsumoto and Okadal . 120021 ). Also, when the mean deviation from the precise 



balance is and the fluctuation is finite, p atterns can be stored at a thermody- 



namic limit ((Matsumoto and Okada . 



2003). This implies that when the balance 



is maintained on average, patterns can be stored. We also showed that when 
the mean is not 0, patterns cannot be stored since a cross-talk noise diverges at 
a thermodynamic limit. However, previous work using computer simulation has 



shown t hat the stored limit eye 



is finite (M unro and Hernandez . 



e is stably retrieved w hen the number of neurons 



2000; 



Yoshioka . 



2002). In the brain, the number 



of neurons is considered to be finite. Therefore, it is important to discuss this 
situation, although no analytical works regarding this has been reported. The 
purpose of the work reported in this paper was to analytically investigate the re- 
trieval properties when the number of neurons is finite. We found that a network 
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containing a finite number of neurons becomes robust regarding this deviation 
from a precise balance between the LTP and LTD of STDP. Since the number 
of neurons in the brain is considered to be finite, our results might apply to the 
brain. 

We introduce our model for the storage of spatio-temporal patterns in section 
2. In section 3, we analytically discuss our model as it applies when the balance 
between the LTP and LTD of STDP is precisely maintained and when it is not 
precisely maintained, and in section 4 we investigate the properties of our model. 
In section 5, we summarize this paper. The Appendix presents a detailed deriva- 
tion of the model's dynamical equations of the model when the balance is precisely 
maintained. 



The model contains N binary neurons with reciprocal connections. Each neuron 
takes a binary value: {0, 1}. We define discrete time steps and the following rule 
for synchronous updating: 



where Xi(t) is the state of the i-th neuron at time t, Ui(t) is the internal potential of 
that neuron, and 9 is a uniform threshold. If the i-th neuron fires at t, its state is 
Xi(t) = 1; otherwise, Xi(t) = 0. Jij is the synaptic weight from the j-th neuron to 



2 Model 



N 




(2) 



(1) 



(3) 
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the i-th neuron. Each element £f of the /i-th memory pattern £ M = (£f , i " • , Civ) 
is generated independently by 

Prob[£f = 1] = 1 - Prob^f = 0] = /. (4) 

The expectation of £ M is E[£f] = /; thus, / is considered to be the mean firing 

rate of the memory pattern. The memory pattern is sparse when / — > 0, and this 

coding scheme is called sparse coding. 

The synaptic weight J^- follows the form of synaptic plasticity, which depends 

on the difference in spike times between the i-th (post-) and j-th (pre-) neurons. 

This difference determines whether LTP or LTD is induced. Such a learning 

rule is called spike-timing-dependent plasticity (STDP). Biological experimental 

findings have shown that LTP or LTD is induced when the differ ence between 

I 



the p re- and post-synaptic spike times falls within about 20 ms (IZhang et al. 



1998) (Figure ^a)). There are also other types of STDP ((Abbott and Nelson 



20001 ) . The STDP typ e 



(e.g., 



Song and Abbottl fj200lh : 



of F ig ure Ufa) has often b een u sed in theoretical work 



Rao and Seinowskil ([20001 ) ). We transformed this 



type of window function into the time window shown in Figure ^b). This time 
window shows that LTP is induced when the j-th neuron fires one time step before 
the i-th. neuron fires, i.e., £f + & = 1, while LTD is induced when the j-th neuron 
fires one time step after the i-th neuron fires, i.e., £f ~ £j* = 1. We define a single 
time step in equations (^GIJ) as 20 ms, and any duration of less than 20 ms is 
ignored. A synaptic plasticity obeying this time window is described by 

Aj i , = e +1 e;-( 1 +^r 1 ^ ( 5 ) 



Impact of deviation from precise balance of STDP 7 
After memory patterns are embedded by STDP, the synaptic weight is written as 

where the value ^jA^n is a scaling factor and the number of memory patterns is 
p = aN where a is defined as the loading rate. If both the i-th neuron and the 
j-th neuron fire simultaneously, neither LTP nor LTD occurs because equation 
does not include the term of £f£^. Since the relative LTP and LTD magnitudes 
are more critical than the absolute magnitudes, the LTD magnitude changes while 
the LTP magnitude and the time duration are fixed, e denotes deviation from a 
precise balance. When e = 0, the balance is precisely ma i ntaine d and this model 



is equivalent to the previous one ([Matsumoto and Okada 



20021 ) . A sequence of p 



memory patterns is stored by STDP: £ x — > £ 2 — ► ■ • • — ► £ p — > ^ —>■ ■ ■ -. In other 
words, £ x is retrieved at t = 1, £ 2 is retrieved at t = 2, and ^ is retrieved at 
t = p + 1. There is a critical value ac of the loading rate, so that a loading rate 
higher than a c causes retrieval of the pattern sequence to become unstable. «c 
is therefore called the storage capacity. 



3 Theory 



3.1 The Case where Balance is Precisely Maintained 



Here, we briefly discuss the case where the balance is prec i sely m aintained, which 



we previously discussed in detail (|Matsumoto and Okada . 



2002). This discussion 



will help explain the case where the balance is not precisely maintained. 

First, we consider a simple situation where there are very few memory patterns 
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relative to the number of neurons; i.e., p ~ 0(1). Let the state at time t be 
equivalent to the t-th memory pattern: x(t) = £*. The internal potential Ui(t) of 
equation (JTJ) is then given by 



Ui(t) depends on two independent random variables, and . The first term 
£* + is the signal term for the recall of the pattern which is intended to be 
retrieved at time t + and the second term £|~ may interfere with the retrieval 
of According to equation (| 7| ). iti(i) takes a value of 0, —1 or +1. The 

probability distribution of Ui(t) is given by Figure Eta); i.e., 



Probst)) = (f-f)5(u l (t)-l) + (l-2f+2f)5(u t (t)) + (f-f 2 )5(u l (t) + l). (8) 

If the threshold 9 is set between and +1, the probability of Ui(t) = is 1 — 2f + 
2f 2 , that of Ui(t) = +1 is / — f 2 , and that of U{(t) = — 1 is / — f 2 . The overlap 
between the state x(t+ 1) and the memory pattern is given by 



At a sparse limit, the overlap m t+1 (t + 1) approaches 1. In other words, the 
memory pattern is retrievable. 

Next, we consider the case where there is a large number of memory patterns, 
i.e., p ~ O(N). The z-th neuronal internal potential Ui(t) at time t is given using 
the periodic boundary condition of £f +1 = Q and £° = £f by 



t-i 



(7) 




8=1 





A' 



P 



(10) 



(3 



t+1 



(ii) 




(12) 
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The first term in equation (jllj) is the signal term for the recall of pattern 
The second term is the cross-talk noise term that represents the contributions 
of non-target patterns other than and prevents the retrieval of the target 

pattern The cross-talk noise is assumed to obey a Gaussian distribution 

with mean and time-dependent variance cr 2 (t). We derive the recursive equation 
for the overlap m*(i) between the state x(t) and the target patte r n by using 



the method of the statistica 



1995; 



Matsumoto and Okada . 



neurodynamics (|Amari and Maginu . 



1988; 



Okada . 



2002) to calculate the overlap m*(t). The recursive 



equation is given by 
m\t) 

where d>n 



1-2/ 



-erti^o 



1 - / / 

— erf (0i ) + -erf(0 2 ), 



(13) 



include a(t — 1), m*(t) depends on a(t). The recursive equation for a(t) is 



m t - 1 (t-l)+9 



. Since 



>i, and 



v 2 (t) = 2(a+i)C {a+1) aq(t - a) JJ U 2 {t - b + 1). 



(14) 



a=0 



6=1 



The recursive equations for U (t) and q(t) are given by 
1 



U(t) 



2vra(t - 1) 



{(1 -2f + 2/>"*° + /(l - /)(e-^ + e"**)}, 



(15) 



and 



q(t) = |(l-(l-2/ + 2/ 2 )erf(0 o ) - /(l - /)(erf(0 1 ) + erf(0 2 ))) , (16) 

where erf(y) = ^ / W exp (-u 2 )du, b C a = aH ^[ a y , a! = a x (a - 1) x • • • x 1. 
The details of this derivation are given in the Appendix. To obtain the storage 
capacity ac, we calculate the overlap in the steady state. We set the initial state 
of the network at the first memory pattern: £c(l) = £ 1 . Setting the initial values at 
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m 1 (l) = 1, cr 2 (l) = 2a f, U(l) = 0, and g(l) = / and using the recursive equations 
(13-16), we obtain the overlap in the steady state. There is a critical value «c of 
the loading rate, so that the loading rate higher than «p causes retrieval of the 
pattern sequence to become unstable. In computer simulation, when the overlap 
in the steady state is smaller than 0.5, the critical loading rate is regarded as the 
storage capacity ate- 

3.2 The Case Where Balance is not Precisely Maintained 

The case where balance is not precisely maintained is difficult to treat because the 
mean of the cross-talk noise is not 0. However, previous work using computer sim- 



ulations has shown that the stored limit cycle is s 



of neurons is finite f Munro and Hernandez 



t ably retrieved w hen the number 



2000; 



Yoshioka . 



2002). However, this 



mathematical mechanism has remained unknown. To investigate this unknown 
mechanism, we will use the theory derived in section 13.11 

First, let the state at time t be equivalent to the £-th memory pattern: x(t) = 
£*. The z-th neuronal internal potential Ui(t) at time t is then given using the 
periodic boundary condition of £f +1 = Q and £° = £f by 

1 N p 

m{t) = £ - (1 + e)^) Xj (t) (17) 

^ N p 

= Nf(l - /) _ ^ }^i x i® 

N p 



w - /) ^ ^ 



where Ci — ~ f ■ Here, we evaluate the second term; that is, the compen- 



sation term in equation (|18p. The average of this term is j^eNq(t) where 
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= = / when x(t) = £*. A typical value of this term is 0.19 

for iV = 5000, a = 0.067, e = 0.05 and / = 0.1, which are typical values used 
in the following analysis. This value is small enough for the overlap m t (t) — 1. 
The fluctuation in the compensation term can be eliminated since the order of the 
variance of this term is smaller than that of the average. Therefore, the average 
term of the compensation term adds to the signal term in equation ([lip. 

Next, we will discuss a mechanism that enables memory patterns to be re- 
trieved in the small e and finite N case. The internal potential with x(t) = is 
given by 

of 2 

«,(*) = e* +1 - r 1 - ^jeN. (19) 

When e ^ 0, each distribution of Ui(t) shifts by fzj^N as shown in Figure I2fb). 
For the typical case at iV = 5000, a = 0.067, e = 0.05, / = 0.1, and = 0.52, 
each distribution of Ui(t) would shift left by 0.19. Since the threshold is 0.52, the 
probability of Ui(t + 1) = +1 is little changed. In other words, for the small e 
and finite iV case, retrieval of the next pattern is successful. Otherwise, the 
retrieval fails. 

Since the average term of the compensation term adds to the signal term in 
equation (jTTjl . the i-th neuronal internal potential at time t is given by 

Ui (t) = - r 1 )™^) + - V"(*) - t=7&)- ( 2 °) 

Here, the first term is the signal term and the second is the cross-talk noise term. 
The third term is the compensation term. Statistical neurodynamics enables us 
to derive dynamical equations. These equations are almost the same as those 
in section 13.11 other than O , 4>i an d 4>2- In 4>o, 4>i an d 02, # is replaced by 



Impact of deviation from precise balance of STDP 12 



9 + s0 q (t-i). 
4 Results 

We used statistical neurodynamics and computer simulation to investigate the 
properties of our model and examine its behavior. 

Figure 01 shows the dependence of the overlap m t {t) on the loading rate a when 
the mean firing rate of the memory pattern is / = 0.1 and the threshold (optimized 
to maximize the storage capacity) is 6 = 0.52. The solid line denotes the steady- 
state values of the overlap m*(i) at e = 0.0 in (a) and e = 0.05 in (b). We obtained 
m*(t) by setting the initial state of the network at the first memory pattern: 
£c(l) = £ 1 . The storage capacity is «c = 0.27 in (a) and «c = 0.067 in (b). The 
data points and error bars (with the former indicating median values and the latter 
indicating 1/4 and 3/4 deviations) show the computer simulation results from 11 
trials with 5000 neurons (N = 5000). The discrepancy between the values of mc 



obtained from the computer simulations and the analytical results was caused 



by 



2002). 



the finite size effect of the computer simulations (|Matsumoto and Okadal 

Figure Ufa) shows the storage capacity «c as a function of e. In the computer 
simulaton, when the overlap in the steady state is smaller than 0.5, the critical 
loading rate a is regarded as the storage capacity ate- The data points and error 
bars (with the former indicating mean values and the latter indicating standard 
deviations) show the computer simulation results from 10 trials at / = 0.1, 6 = 
0.52, and N = 3000 (□) or N = 5000 (o). The solid line shows the theoretical 
results at iV = 3000, while the dashed line and the gray solid line show the results 
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at N — 5000 and iV = 100000, respectively. As |e| increased, otc decreased. When 
e = 0.5, the storage capacities for N = 3000 and A = 5000 were 0.017 and 0.011, 
respectively. However, the storage capacity for A = 100000 was 0. In other words, 
when the number of neurons is finite, takes a finite value. These computer 
simulation results are consistent with the theoretical results at small e. Figure 
EJb) shows the maximum number of storable memory patterns, p ma x — cx-cN , as 
a function of e. This is replot of Figure Efa). The solid line shows the theoretical 
results at iV = 3000, while the dashed line and the gray solid line show the results 
at iV = 5000 and A = 100000, respectively. As |e| increased, actual number of 
storable memory patterns decreased. 

Figure El shows otc as a function of e. The solid line shows the theoretical results 
at / = 0.1 and 6 = 0.52, while the dashed line shows log 10 ac = — log 10 Ae + 1.435 
at A = 5000, / = 0.1, and 9 = 0.52. This figure shows that ac converged to as 
the order of (i.e., O(-j^)) at a large Ae limit. 

5 Summary and Discussion 

We have investigated, using an associative memory network, how the balance be- 
tween the LTP and LTD of STDP affects the retrieval of spatio-temporal patterns. 
Other authors have numerically investigated the impact of an imbalance between 
LTP and LTD on network properties. We have analytically investigated the re- 
trieval properties by applying statistical neurodynamics. In the case where the 
value of LTD deviates from a precise balance the stored limit cycle using STDP 
is unstable at the thermodynamic limit. However, the stored limit cycle is sta- 
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ble when the number of neurons is finite. Moreover, computer simulation results 
were consistent with the theoretical results at small e. Thus, a network contain- 
ing a finite number of neurons becomes robust against deviation from a precise 
LTP/LTD balance. Since the number of neurons in the brain is considered to 
be finite, our results might be applicable to the brain. Furthermore, the storage 
capacity converges to as O(j^) at a large Ne limit. 
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Appendix. Derivation of Recursive Equations when 
Balance is Precisely Maintained 

In this appendix, we derive recursive equations for the overlap m*(t) when the 
balance between the LTP and LTD of STDP is precisely maintained. We divide 
an internal potential Ui(t) at time t into two parts, a signal term for a retrieval of 
a target pattern and a cross-talk noise term that represents the contributions from 
non-target patterns and prevents the target pattern from being retrieved. Ui(t) is 
derived from equation (JBJ) and the periodic boundary conditions of £f +1 = Q and 
C° = £f m the following way: 

1 N p 

«*(*) = Nf(1 _ f) E E^tf - r 1 ^*) (2i) 



eg* -rvw +*(*). ( 22 ) 



where Zj(t) is given by 



^) = E(^ +1 -erV M (o, (23) 

and m M (t) is the overlap between £ M and £c(t) and which is given by 

1 - 

m»(t) = Nf{1 _ f) ~ f)*i<t)- (24) 

The first term in equation (22) is the signal term and the second term is the 
cross-talk noise term. Since Xi(t) in equation (24) depends on £f , the distribution 
of the cross-talk noise term Zi(t) is unknown. The dependence on ^ is extracted 
from Xi(t) using the Taylor expansion. 

At time t, the pattern is designed to be retrieved. Therefore, we can 
assume that m*(i) is order of 1 with respect to iV (i.e., m*(t) ~ 0(1)) and 
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m»(t){u. ^ t) is order of l/y/N with respect to JV (i.e., m»(t) ~ 0(1/VN)). 
Since m^(t) ~ 0(1/ VN), m M_1 and m M+1 are order of 1/y/N with respect to iV. 
At a thermodynamic limit, iV — > oo, m M_1 and m M+1 are small. To extract the 
dependence on ^ from Xj(t), the i-th neuronal state at time t + 1 is transformed: 



x t (t + 1) = F(J2(^ +1 - £r>^) - *) (25) 

=^(E(er +1 -r i )^w-^) 
+er(^- i (o - ^ +i (t))nE(£? +1 - ervw - *) (26) 

= xS M) (t + 1) + ^(m^ 1 (i) - m^t))*'/" } (t + 1), (27) 

where £f = £f — /, xf\t + 1) does not include £f , and x'^\t) is the differential 
of Using this relationship, we derive the cross-talk noise at time t: 

^w=E(^r 1 -^rv /i (*) (28) 

p iV 



^/(i - /) 

+ E ^(0(er +1 ^ _1 (^ - 1) - ^r 1 ™*-^ - 1) + er^ +i (t - 1))(29) 



where U(t) = E[x'^\t)\. Therefore, the square of Zi(t) is given by 
/ i \ 2 p N 



+ E ^W 2 ^ 1 ^ 1 ^ - 1) - 2#- 1 m"- 1 (f - 1) 

+er 1 m" +i (* - 1)) 2 (so) 

= (l 2 + (-lf)aq(t) + (l 2 + (-2) 2 + l 2 )aq(t - l)U 2 (t) 

+ (1 2 + (-3) 2 + 3 2 + {-lf )aq{t - 2)U 2 {t)U 2 {t -!) + ••• (31) 
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t a 

= 2(a+i)C {a+1) aq(t - a) J] U 2 {t - b + 1), (32) 

a=0 6=1 

where p = aN, q(t) = ^2^ =1 (x^\t)) 2 , f,C a = a \^L a y_ and a ' is a factorial with 
positive integer a. We applied the relationship ^a =0 (6C a ) 2 = 2bCb in this deriva- 
tion. At a thermodynamic limit, N — > oo, m^(t) tends to be deterministic. There- 
fore, a^(i) is independent of £f. This enables us to assume that the cross-talk 
noise term obeys a Gaussian distribution with a mean of and time-dependent 
variance cr 2 (t): E[zj(i)] = 0, E[(^(t)) 2 ] = o- 2 (t). Since the distribution of the cross- 
talk noise term is known, recursive equations for m*(t) and cr 2 (t) are obtained. We 
get the recursive equation for a 2 (t): 

t a 

v\t) = 2(a+i)C {a+1) aq(t - a) J] U 2 (t -6+1). (33) 

a=0 6=1 

The overlap between the state x(t) and the target pattern £* is given by 

1 N 1 N 

m\t) = Jrw - T) Ytf - /,,,-(/, = Ytf - f)F{um (34) 

1 N 

= Nf{1 _ f) - - ^" 2 )"»* _1 (* - 1) + *(* - 1) - (35) 

Since ttj(£) is independent and identical distribution (i.i.d.), by the law of large 
numbers, we can replace the average over i by an average over the memory patterns 
£ M and the Gaussian noise term z ~ A/"(0, <x 2 ). Then, the recursive equation for 
the overlap m*(t) is transformed: 



xFae-e-^-'it-V + z-e))) (36) 

1 1 



f(i-f)V^J- 



/ dze-^(((e-f) 
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xF((e - - 1) + *(t - l)~z - &))) (37) 

'' 2/ -erf(0 o ) - ^erf(0O + 4rf(0 2 ), (38) 



2 vru/ 2 vr y 2 

where erf(y) = £ JJ exp (-»')<!«, O = , = ^HF> = ^Sl^f > 

and ((■)) denotes an average over the memory pattern Since x'^t) — x'^\t) ~ 
0(-j=) and the thermodynamic limit iV — > 00 is considered, x'^\i) = x'^t). Using 
this relationship, we derive £/(£): 

TV TV 

f/ w=^E^ ) w = ^E^w ( 39 ) 

i=l i=l 

' dze-^((F'((^-^- 2 )m*- 1 (t-l)+<T(t- 1)^-0))) (40) 



'' {(1 - 2/ + 2/V* 8 + /(i _ /)(e -*? + e -*g)}. (41) 



V27r(7(t - 1) 

Since Xi(t)—x^\t) ~ 0(^=) and N — > 00, = Using this relationship, 

we derive g(t): 

N N 

^)=^E(^ ) w) 2 = ^E^w) 2 ( 42 ) 

1=1 1=1 

/ rfze"- ((F 2 ((e* - f*- 2 )™*- 1 ^ - 1) + <r(t - \)z - 6))) (43) 

«/ —00 



i (1 - (1 - 2/ + 2/ 2 )erf(0 o ) - 7(1 - /)(erf(0 1 ) + erf(0 2 ))) . (44) 



Figure legends 



Figure 1: The time windo w of spike-timing-d ependent plasticity, (a): Results of 



the biological experiment (Zh ang et al 



1998). (b): STDP in our model. LTP 



occurs when the j-th neuron fires one time step before the i-th neuron. LTD 
occurs when the j-th neuron fires one time step after the i-th neuron. 

Figure 2: Probability distribution of the i-th neuronal potential Ui(t) at time t. 
(a)The balance between the LTP and LTD of STDP is precisely maintained; i.e., 
e = 0. (b)The balance is not precisely maintained; i.e., e ^ 0. Each distribution 
shifts by j^eN. 

Figure 3: The solid line shows the overlap in the steady state when the firing 
rate of stored patterns is 0.1 at e = 0.0 in (a) and e = 0.05 in (b). The storage 
capacity is 0.27 in (a) and 0.067 in (b). In both figures, the threshold is 0.52 and 
the number of neurons is 5000. The data points indicate the median values and 
the ends of the error bars indicate 1/4 and 3/4 derivations, respectively, from 11 
computer simulation trials. 



Figure 4: (a)The storage capacity «c and (b)the maximum number of storable 
memory patterns, p max = olcN , (b) as a function of e. The data points and error 
bars (with the former indicating the mean values and latter indicating standard 
deviations) in (a) show computer simulation results from 10 trials at iV = 3000 
(□) or N = 5000 (o). The solid line, dashed line and gray solid line show the 
theoretical results at N = 3000, N = 5000, and N = 100000, respectively. All 
results were obtained at / = 0.1 and 9 = 0.52. As |e| increased, «c and p ma x 
decreased. 

Storage capacity ac as a function of e at iV = 5000, / = 0.1, and 
The solid line shows the theoretical results while the dashed line shows 
- — log 10 iVe + 1.435. a c converged to as 0(^-) at a large iVe limit. 



Figure 5: 
9 = 0.52. 

logio a c ~- 
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